Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎨 EFS Guardian: adding size monitoring #6502

Merged
merged 18 commits into from
Oct 10, 2024
Merged

🎨 EFS Guardian: adding size monitoring #6502

merged 18 commits into from
Oct 10, 2024

Conversation

matusdrobuliak66
Copy link
Contributor

@matusdrobuliak66 matusdrobuliak66 commented Oct 8, 2024 •
edited
Loading

What do these changes do?

  • ✨ Adding size monitoring. EFS Guardian subscribes to a queue that listens for heartbeats when a dynamic service is running. Each minute, a heartbeat triggers a monitoring function that checks the current size. If the size exceeds the threshold, write permissions on all files are removed.
  • ✨ Prepare the groundwork for a background task that will handle removing data from EFS for projects that have not been accessed for some time.

Related issue/s

How to test

Dev-ops checklist

@matusdrobuliak66 matusdrobuliak66 self-assigned this Oct 8, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 8, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 85.26316% with 28 lines in your changes missing coverage. Please review.

Project coverage is 88.1%. Comparing base (cafbf96) to head (bd5eb5e).
Report is 626 commits behind head on master.

Files with missing lines Patch % Lines
..._service_efs_guardian/services/process_messages.py 35.7% 18 Missing ⚠️
...service_efs_guardian/services/efs_manager_utils.py 73.9% 6 Missing ⚠️
...ls-library/src/models_library/rabbitmq_messages.py 77.7% 1 Missing and 1 partial ⚠️
...ce_efs_guardian/services/process_messages_setup.py 94.8% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6502      +/-   ##
=========================================
+ Coverage    84.5%   88.1%    +3.5%     
=========================================
  Files          10    1546    +1536     
  Lines         214   63324   +63110     
  Branches       25    2059    +2034     
=========================================
+ Hits          181   55807   +55626     
- Misses         23    7199    +7176     
- Partials       10     318     +308
Flag Coverage Ξ”
integrationtests 64.7% <42.8%> (?)
unittests 86.1% <85.2%> (+1.5%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Ξ”
...c/simcore_service_dynamic_sidecar/core/rabbitmq.py 100.0% <100.0%> (ΓΈ)
...dynamic_sidecar/modules/resource_tracking/_core.py 96.2% <100.0%> (ΓΈ)
...c/simcore_service_efs_guardian/core/application.py 97.2% <100.0%> (ΓΈ)
.../src/simcore_service_efs_guardian/core/settings.py 100.0% <100.0%> (ΓΈ)
..._service_efs_guardian/services/background_tasks.py 100.0% <100.0%> (ΓΈ)
...ce_efs_guardian/services/background_tasks_setup.py 100.0% <100.0%> (ΓΈ)
...mcore_service_efs_guardian/services/efs_manager.py 100.0% <100.0%> (ΓΈ)
...ore_service_efs_guardian/services/modules/redis.py 100.0% <100.0%> (ΓΈ)
...ls-library/src/models_library/rabbitmq_messages.py 89.4% <77.7%> (ΓΈ)
...ce_efs_guardian/services/process_messages_setup.py 94.8% <94.8%> (ΓΈ)
... and 2 more

... and 1484 files with indirect coverage changes

@matusdrobuliak66 matusdrobuliak66 changed the title WIP: 🎨 EFS Guardian: adding monitoring functionality 🎨 EFS Guardian: adding monitoring functionality Oct 9, 2024
@matusdrobuliak66 matusdrobuliak66 marked this pull request as ready for review October 9, 2024 14:16
@matusdrobuliak66 matusdrobuliak66 changed the title 🎨 EFS Guardian: adding monitoring functionality 🎨 EFS Guardian: adding size monitoring Oct 9, 2024
GitHK
GitHK approved these changes Oct 10, 2024
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, please check some of my suggestions

services/efs-guardian/src/simcore_service_efs_guardian/services/efs_manager_utils.py Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/process_messages.py Outdated Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/process_messages_setup.py Outdated Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/background_tasks.py Show resolved Hide resolved
pcrespov
pcrespov reviewed Oct 10, 2024
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR

  • the dy-sidecar notifies DynamicServiceRunningMessage and the efs-guardian listens
  • efs-guardian:
    • prepares background modules (still not fully implemented) which needs redis for distributed lock mechanism
    • EfsManager operates on EFS mounted folder. I understand is a "single drive" and you only manipulate one project/node at a time?
    • process-messages reacts to a service running. If size is over threshold, it remove write premssions. Uses redis as a distributed lock mechanims here as well.

Generally, I think you should give a deeper thought about including good testing and error handling.

services/efs-guardian/src/simcore_service_efs_guardian/services/modules/redis.py Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/background_tasks_setup.py Outdated Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/background_tasks_setup.py Outdated Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/background_tasks.py Outdated Show resolved Hide resolved
services/efs-guardian/src/simcore_service_efs_guardian/services/efs_manager_utils.py Outdated Show resolved Hide resolved
services/efs-guardian/tests/unit/test_efs_manager.py Outdated Show resolved Hide resolved
services/efs-guardian/tests/unit/test_efs_manager.py Outdated Show resolved Hide resolved
services/efs-guardian/tests/unit/test_efs_manager.py Outdated Show resolved Hide resolved
services/efs-guardian/tests/unit/test_efs_manager.py Show resolved Hide resolved
services/efs-guardian/tests/unit/test_efs_manager.py Show resolved Hide resolved
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@matusdrobuliak66 matusdrobuliak66 merged commit f3e838b into ITISFoundation:master Oct 10, 2024
57 checks passed
@matusdrobuliak66 matusdrobuliak66 deleted the improve-efs-2 branch October 10, 2024 13:40
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-simcore that referenced this pull request Oct 14, 2024
@matusdrobuliak66 @mrnicegyu11
🎨 EFS Guardian: adding size monitoring (ITISFoundation#6502)
4d27c32
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Oct 15, 2024
Closed
18 tasks
This was referenced Oct 23, 2024
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@pcrespov pcrespov pcrespov approved these changes

@GitHK GitHK GitHK approved these changes

@sanderegg sanderegg Awaiting requested review from sanderegg sanderegg is a code owner

@mrnicegyu11 mrnicegyu11 Awaiting requested review from mrnicegyu11 mrnicegyu11 is a code owner

Assignees

@matusdrobuliak66 matusdrobuliak66

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@matusdrobuliak66 @GitHK @pcrespov @YuryHrytsuk